Bug 1806009: Backport ovn-kubernetes-related changes to 4.3 by danwinship · Pull Request #483 · openshift/cluster-network-operator

danwinship · 2020-02-18T16:57:18Z

Backport of all as-yet-unbackported ovn-kubernetes-related fixes to 4.3

Includes:

ovn-kube: add pod disruption budget, readiness check #414 ovn-kube: add pod disruption budget, readiness check
osdn, ovnk: add publishNotReadyAddresses to metrics services #427 ovnk: add publishNotReadyAddresses to metrics services
Use app label instead of name on ovnkube-master ds #419 + ovn: use app label instead of name for master metrics and quorum guard #435 Use app label instead of name on ovnkube-master ds, etc
ovn: fix master control plane job name #436 ovn: fix master control plane job name
OVN: use config file via ConfigMap rather than environment variables #333 + Remove some unwanted logging. Fix an error message #437 + ovn: container env variables need quotes #440 + Bug 1802596: Remove erroneous ovnkube node command-line args #473 OVN: use config file via ConfigMap rather than environment variables
ovn custom geneve port number #341 ovn custom geneve port number
Support ovn2.12 #439 + Bug 1798191: Bump to ovn2.12 #445 - Support ovn2.12
Bug 1796718: ovn: use ovs-appctl ovsdb-server/sync-status for NB/SB DB readiness #461 ovn: use ovs-appctl ovsdb-server/sync-status for NB/SB DB readiness
Bug 1796562: sdn/ovn: don't allow ovs-vswitchd to clean up datapath flows on exit #460 ovn: don't allow ovs-vswitchd to clean up datapath flows on exit

Does not include:

anything SDN-related (not even the SDN-related parts of osdn, ovnk: add publishNotReadyAddresses to metrics services #427 and Bug 1796562: sdn/ovn: don't allow ovs-vswitchd to clean up datapath flows on exit #460)
anything Kuryr-related
anything unmanaged-OVS-related
anything IPv6-related since that had all already been backported
the as-yet-unmerged-anyway Remove support for ovn2.11 #467 (remove support for ovn2.11)

Add a PodDisruptionBudget to protect the raft quorum. Configure a readines probe for the DBs: ovsdb raft only opens its port once it has a raft consensus. Utilize that. Also, add a TerminationMessagePolicy. (cherry picked from commit b2f02b3)

Since these services are just used to configure Prometheus scrape targets, rather than load-balance traffic, we should not remove not-ready addresses from them. (cherry picked from commit f9cdcff)

(cherry picked from commit 484aa15) (cherry picked from commit ab9f0fc)

It's ovnkube-master not ovn-kubernetes-master. (cherry picked from commit b767fd7)

Configure ovnkube using a config file passed by the ovnkube-config configMap and mounted into the container. (cherry picked from commit eb3b64d) (cherry picked from commit 4bda7b7) (cherry picked from commit a520900) (cherry picked from commit 79bb2b8) (cherry picked from commit ec6812f)

SDN-494 - ovn custom geneve port number. https://jira.coreos.com/browse/SDN-494 Signed-off-by: Phil Cameron <pcameron@redhat.com> (cherry picked from commit f1b6046)

openshift-ci-robot · 2020-02-18T16:57:21Z

@danwinship: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

Backport ovn-kubernetes-related changes to 4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

danwinship · 2020-02-18T16:59:13Z

/cc @dcbw @pecameron @rcarrillocruz
/test e2e-gcp-ovn
/test e2e-aws-ovn

russellb · 2020-02-19T15:53:53Z

I tried an install with this and got:

level=error msg="Cluster operator network Degraded is True with ApplyOperatorConfig: Error while updating operator configuration: could not apply (apps/v1, Kind=DaemonSet) openshift-ovn-kubernetes/ovnkube-node: could not create (apps/v1, Kind=DaemonSet) openshift-ovn-kubernetes/ovnkube-node: DaemonSet.apps \"ovnkube-node\" is invalid: spec.template.spec.containers[0].volumeMounts[4].name: Not found: \"old-openvswitch-database\""
level=info msg="Cluster operator network Progressing is True with Deploying: DaemonSet \"openshift-multus/multus-admission-controller\" is not yet scheduled on any nodes\nWaiting for DaemonSet \"openshift-ovn-kubernetes/ovnkube-node\" to be created"
level=info msg="Cluster operator network Available is False with Startup: The network is starting up"

ovn2.12 places the ovn-ctl script in a different directory than earlier versions. This uses the correct path for the installed version. (cherry picked from commit 3fca003) (cherry picked from commit 2711b32)

russellb · 2020-02-19T18:13:19Z

Tested again, this time ovnkube-node is failing, because it seems it got configured with an IPv4 subnet.

time="2020-02-19T18:09:26Z" level=fatal msg="invalid hostsubnet found for node master-2.ostest.test.metalkube.org: invalid CIDR address: 10.129.0.0/64"

danwinship · 2020-02-19T18:15:12Z

yeah, I guess the change to use the config file requires the corresponding ovn-kubernetes fix to not accidentally replace the config options with the defaults

russellb · 2020-02-19T18:23:53Z

yeah, I guess the change to use the config file requires the corresponding ovn-kubernetes fix to not accidentally replace the config options with the defaults

upstream: ovn-kubernetes/ovn-kubernetes#1044

openshift master: openshift/ovn-kubernetes#93

pecameron · 2020-02-21T16:47:36Z

@danwinship I included #458, #454, and #443. PTAL and see if you think they should be included.

pecameron

/lgtm

pecameron · 2020-02-21T19:23:58Z

e2e-upgrade-gcp has OOMKilled messages in the journals
$ grep OOMKilled e2e-gcp-upgrade/nodes/masters-journal |wc -l
21
$ grep OOMKilled e2e-gcp-upgrade/nodes/workers-journal |wc -l
5

/test e2e-gcp-upgrade

openshift-bot · 2020-02-21T19:41:09Z

/retest

Please review the full test history for this PR and help us cut down flakes.

danwinship · 2020-02-21T19:57:11Z

@danwinship I included #458, #454, and #443. PTAL and see if you think they should be included.

458 only changes sdn-specific stuff
454 is a multus thing which was just to meet new validation requirements for 4.4.
443 is only needed to support the migration tool, which won't be supported in 4.3.

so I don't think any of those are needed

openshift-ci-robot · 2020-02-21T20:03:57Z

@danwinship: This pull request references Bugzilla bug 1806009, which is invalid:

expected dependent Bugzilla bug 1806006 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), but it is NEW instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 1806009: Backport ovn-kubernetes-related changes to 4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

danwinship · 2020-02-21T20:04:26Z

/bugzilla refresh

openshift-ci-robot · 2020-02-21T20:04:33Z

@danwinship: This pull request references Bugzilla bug 1806009, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pecameron · 2020-02-21T21:08:23Z

@danwinship Are you happy with the bumo to ovn2.12 with 2.12 in its current state? @dcbw was talking about backports coming.

danwinship · 2020-02-22T22:55:45Z

The bumping-to-2.12 here is just updating for the changed paths. Further ovn-kubernetes backports won't affect that.

danwinship · 2020-02-22T22:55:57Z

/test e2e-gcp-ovn
/test e2e-aws-ovn

alexanderConstantinescu · 2020-02-24T17:31:31Z

@danwinship : could you add this commit to the list of cherry-picks for this PR (the master PR has been merged now)?

31476cc

russellb · 2020-02-25T18:12:21Z

patch manager note: this is required for Ipv6 support

knobunc · 2020-02-25T18:42:15Z

This depends on openshift/ovn-kubernetes#100

Rather than connecting to the database TCP socket which just spams the database logs with failed SSL connection messages. (cherry picked from commit 876e67a) (cherry picked from commit 31476cc)

When terminated gracefully, either via 'ovs-ctl stop' or TERM, ovs-vswitchd cleans up datapath flows. This prevents any existing flows from continuing to work, which kinda defeats the purpose of a seamless upgrade. The preStop hook is removed because it is redundant with the TERM trap of the container's main process bash script. (cherry picked from commit 2f543fa)

danwinship · 2020-02-25T20:36:45Z

@danwinship : could you add this commit to the list of cherry-picks for this PR (the master PR has been merged now)?

done (squashed with the original patch from dcbw)

russellb · 2020-02-25T21:05:13Z

/lgtm

openshift-ci-robot · 2020-02-25T21:06:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danwinship, pecameron, russellb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [danwinship]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-02-25T23:07:30Z

/retest

Please review the full test history for this PR and help us cut down flakes.

eparis · 2020-02-26T23:56:05Z

/retest
level=error msg="Error: Error applying IAM policy to project \"openshift-gce-devel-ci\": Too many conflicts. Latest error: Error setting IAM policy for project \"openshift-gce-devel-ci\": googleapi: Error 409: There were concurrent policy changes. Please retry the whole read-modify-write with exponential backoff., aborted" Is not this PR

openshift-bot · 2020-02-27T00:38:22Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-27T01:43:11Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-02-27T01:56:12Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot · 2020-02-27T03:11:57Z

@danwinship: All pull requests linked via external trackers have merged. Bugzilla bug 1806009 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1806009: Backport ovn-kubernetes-related changes to 4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

squeed and others added 6 commits February 18, 2020 11:18

ovnk: add publishNotReadyAddresses to metrics services

d815e61

Since these services are just used to configure Prometheus scrape targets, rather than load-balance traffic, we should not remove not-ready addresses from them. (cherry picked from commit f9cdcff)

Use app label instead of name on ovnkube-master ds, etc

0dae2c1

(cherry picked from commit 484aa15) (cherry picked from commit ab9f0fc)

ovn: fix master control plane job name

f8bdc27

It's ovnkube-master not ovn-kubernetes-master. (cherry picked from commit b767fd7)

ovn custom geneve port number

96aa64c

SDN-494 - ovn custom geneve port number. https://jira.coreos.com/browse/SDN-494 Signed-off-by: Phil Cameron <pcameron@redhat.com> (cherry picked from commit f1b6046)

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 18, 2020

openshift-ci-robot requested review from rcarrillocruz and squeed February 18, 2020 16:58

openshift-ci-robot requested review from dcbw and pecameron February 18, 2020 16:59

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 18, 2020

Support ovn2.12

92c6593

ovn2.12 places the ovn-ctl script in a different directory than earlier versions. This uses the correct path for the installed version. (cherry picked from commit 3fca003) (cherry picked from commit 2711b32)

danwinship force-pushed the ovn-kube-rebase branch from c4b4959 to 4e632b1 Compare February 19, 2020 16:42

openshift-ci-robot assigned pecameron Feb 21, 2020

pecameron reviewed Feb 21, 2020

View reviewed changes

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 21, 2020

This was referenced Feb 21, 2020

Bug 1803701: Back-port changing OVN DB readinessProbe to check membership status instead of … #478

Closed

Bug 1801384: [4.3]Bump to ovn2.12 #469

Closed

danwinship changed the title ~~Backport ovn-kubernetes-related changes to 4.3~~ Bug 1806009: Backport ovn-kubernetes-related changes to 4.3 Feb 21, 2020

openshift-ci-robot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Feb 21, 2020

openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Feb 21, 2020

alexanderConstantinescu mentioned this pull request Feb 24, 2020

Bug 1775838: Back-port, OVNKubernetes: introduce OVS anti-selector #492

Merged

dcbw added 2 commits February 25, 2020 15:35

ovn: use ovs-appctl for NB/SB DB readiness

6a15ba6

Rather than connecting to the database TCP socket which just spams the database logs with failed SSL connection messages. (cherry picked from commit 876e67a) (cherry picked from commit 31476cc)

danwinship force-pushed the ovn-kube-rebase branch from 4e632b1 to 65fb42e Compare February 25, 2020 20:36

openshift-ci-robot removed the lgtm Indicates that a PR is ready to be merged. label Feb 25, 2020

openshift-ci-robot assigned russellb Feb 25, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 25, 2020

eparis added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Feb 26, 2020

openshift-merge-robot merged commit 0ccb533 into openshift:release-4.3 Feb 27, 2020

Conversation

danwinship commented Feb 18, 2020

Uh oh!

openshift-ci-robot commented Feb 18, 2020

Uh oh!

danwinship commented Feb 18, 2020

Uh oh!

russellb commented Feb 19, 2020

Uh oh!

russellb commented Feb 19, 2020

Uh oh!

danwinship commented Feb 19, 2020

Uh oh!

russellb commented Feb 19, 2020

Uh oh!

pecameron commented Feb 21, 2020 • edited by danwinship Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pecameron left a comment

Choose a reason for hiding this comment

Uh oh!

pecameron commented Feb 21, 2020

Uh oh!

openshift-bot commented Feb 21, 2020

Uh oh!

danwinship commented Feb 21, 2020

Uh oh!

openshift-ci-robot commented Feb 21, 2020

Uh oh!

danwinship commented Feb 21, 2020

Uh oh!

openshift-ci-robot commented Feb 21, 2020

Uh oh!

pecameron commented Feb 21, 2020

Uh oh!

danwinship commented Feb 22, 2020

Uh oh!

danwinship commented Feb 22, 2020

Uh oh!

alexanderConstantinescu commented Feb 24, 2020

Uh oh!

russellb commented Feb 25, 2020

Uh oh!

knobunc commented Feb 25, 2020

Uh oh!

danwinship commented Feb 25, 2020

Uh oh!

russellb commented Feb 25, 2020

Uh oh!

openshift-ci-robot commented Feb 25, 2020

Uh oh!

openshift-bot commented Feb 25, 2020

Uh oh!

eparis commented Feb 26, 2020

Uh oh!

openshift-bot commented Feb 27, 2020

Uh oh!

openshift-bot commented Feb 27, 2020

Uh oh!

openshift-bot commented Feb 27, 2020

Uh oh!

openshift-ci-robot commented Feb 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

pecameron commented Feb 21, 2020 •

edited by danwinship

Loading